Mystery (partially) solved. I just took a sneak peak at the extended attributes of the .DS_Store file and some have a resource fork (probably the ones that were failing to copy I BET!). So appears to be the same failure we are talking about here: https://developer.apple.com/forums/thread/814076 .DS_Stores on the NAS's with resource forks.. maybe that will help you figure out leaky compression

Interesting and really good information to know! IF this is in fact a compressed file, then just copying the resource fork won't actually work. There's an additional UF_COMPRESSED file flag that needs to be set, which then "marks" the file as compressed so that the system knows to retrieve and decompress the file contents when the file itself is opened. If you read through the previous threads, you can see that this flag is what ends up "hiding" the presence of the resource fork from the rest of the system. However, if it was a compressed file then it should never have been written to the NAS device as a compressed file. All of our copy engines would have either automatically decompressed it (this is what happens if you copy the file without "knowing" about file compression) or noticed that the target DIDN'T support compression... and decompressed the file. "Blame" is definitely the critical factor here. The worst-case scenario I'd be concerned about is that the files were already damaged when they hit the NAS and UF_COMPRESSED was stripped. You then copy the file "properly" (replicating the file as it exists on the NAS) and then get blamed for both the bad copy AND damaging the original (since you were the last one to "touch" it). [...] My personal inclination would probably be to use Carbon, as that gets you closest to "do what the Finder does”, which is the benchmark most developers are concerned with. FYI, you can "fix" these files (by setting UF_COMPRESSED), but that is something the user would need to be involved with, not something you can do automatically[1]. In regards to the potential compression situation, I'm assuming Finder copying doesn't involve the user or set UF_COMPRESSED? Or does it? If it just copies and the file, and that file remains 'unusable' then I guess copyfile's behavior (failing) would be better in that case. Now if there is someone out there just hoarding really old files with resource fork, Finder's behavior is obviously superior. All it takes, as you demonstrated, to get copyfile to fail is to feed it a file with a resource fork. And the open source code indicates that it at least tries to copy it. So I guess I just want to copy the data the user feeds me. Whether the file is already broken due to compression leak etc. that's outside my app's bounds. My app isn't a "data repair tool," it isn't in that genre, so I just am going to have to settle for copying whatever I get from the pasteboard. In the case of a file with a resource fork, I guess I'd hope it is a very old file from ancient times and not a broken file. It'd be great if Finder shared the same copy engine with the rest of us, at least whatever issues would be consistent system wide. But I get it..it probably won't ever happen.

I think the idea you're getting at here is that you should be able to point your copy engine at any point in the file system, have it do the "right" thing based only on the information it's provided by copyfile, not external knowledge (like the full source path). [...] That's a lovely idea, but for better or worse, macOS just does not work that way. [...] but if you want to work with the "full" system... Well, I don't think there's any way to do without grappling with a whole bunch of edge cases. Well I did not directly point copyfile at .DS_Store and tell it to do the thing. I started a recursive copy on a folder, which happens to have a .DS_Store file in it. And on macOS just about every folder the user views in Finder will have one of those hidden in there, unfortunately. The folder I was copying was on a NAS, has like 24 videos in it. I'm doing this intentional slow copy to test everything out. So the copy gets like 98% done and then it needs a diaper change on the .DS_Store file! I'd ignore all .DS_Store errors and, potentially, just skip copying them all together. I have no intention of including .DS_Store in the copy. I am ignoring it now. I'm not really sure if they should even exist and probably shouldn't passively travel in a copy (maybe with the one exception when an advanced user is viewing hidden files and actively puts it in the pasteboard). It's okay with me if I can't do the copy on DS_Store, I guess. I generally want my app to do what the user is trying to do as long as it can't hurt them. My main issue is it doesn't report the reason why I can't do the copy. In fact sometimes I can do the copy but similar to the issue in the other thread it seems that when the .DS_Store is on the NAS, copyfile fails and that may have something to do with it. But I can copy .DS_Store that are on the local drive (perhaps because of the clone option I dk). And yes Finder can copy it, even from the NAS. So I guess if I really wanted to copy it I might be able to dust off that old Carbon File Manager (but I actually don't). Mainly my concern is cryptic failures even though I am primarily dealing with user facing files in reasonable user-facing locations. I'm not really trying to travel to the underworld. In any case if my app doesn't have the right privileges to copy .DS_Store or whatever (which are littered all over the file system) a normal error, even one with no recovery options seems appropriate. I really want to know if it failed because of something else and not permissions (perhaps because the way the data is stored). If there is another file I come across that stores data in a similar way...and the user would reasonably WANT to copy that file, that's actually what I'm more concerned about than .DS_Store.

Just read your reply again more carefully as it is not the end of the day and I'm not tired :) . As the thread up above shows, the primary way to bypass the issue "copyfile" is having is to use the "..namedfork/rsrc" convention, Interesting. I was using xattr API. So just skipping the resource fork during copyfile, then running copyfile again over the src and destination (with .namedfork/rsrc convention) seems to work on my dummy test file. Is that expected to be a reliable workaround? I guess if this actually works reliably I wouldn't have to restart copy from scratch or suppress deprecated API warnings. Or maybe there good reason to start the copy from scratch with the Carbon file manager?

Have you filed a bug on this and, if so, what is the bug number? Not yet, but I do have another thread where I describe the same problem in regards to .DS_Store: https://developer.apple.com/forums/thread/819041 I will post the bug number (most likely there) when I carve out a little time for it. Currently if copyfile fails and errno is not set my code falls back to a kind of 'Unknown Error' which I hate but if the user is counting on my app to perform the copy it's kind of a gamble if the operation doesn't succeed to just hope not critical data was lost. A big part of this is that xattrs aren't actually "safe“— the system doesn't have a documented system for defining how they should be handled, and app/copy engine/file format support for them is inconsistent, so storing critical data in them can end up being a good way to lose critical data. Importantly, that kind of data loss isn't really a "bug" in the same way other kinds of data loss would be. The general "point" of xattrs is to allow the attachment of arbitrary metadata, but part of that contract is that it's not really part of the "core" file data. This was always my understanding of xattrs. The resource fork WAS originally intended as a larger-scale data storage location, which meant repurposing it was substantially less disruptive (and required a lot less work) than creating a new "bulk" xattr would have been, particularly in the context of HFS+ (where they were originally architected and introduced). Yea my concern is if these files are really old the user may be keeping them around for a long time because they consider them to be important. So I wouldn't want to be blamed if they copied the file in my app, discard the original, and the copy is corrupted. Like the Carbon framework, resource forks were before my time though. The answer depends on what you're trying to do. Basically what I'm trying to do is avoid failure. If I can reasonably and gracefully handle the issue and not present an error, that is best. At the same time, I feel a responsibility to the user. If I don't show an error, the copy should have worked and I don't want to silently hide true errors. Another less important need (but still kind of important), is to keep the implementation as simple as possible. I don't want to rewrite copyfile as I already got like three billion lines of code in my app and I have tons of other stuff to tackle. I don't see much return on investment doing that. Basically my workaround currently is if I run into a resource fork in the copyfile callback, I set a flag and skip it. Then when copyfile finishes (if successfully) I just copy the resource fork over from source to destination. If that sounds like a bad idea, let me know and maybe I'll just fallback to the Carbon file manager instead.

Sorry for the double post. But just a quick additional question. Am I better off working around the issue by retrying with the Carbon file manager or should I just skip the resource fork in the copyfile callback, then copy it over myself at the end?

I'm experiencing this issue. Copy file works locally for these files but fails when coming from a NAS. Works from Finder in both cases, as previously described above. copyfile does not seem to provide a usable error. The callback is invoked twice and I get : what == COPYFILE_COPY_XATTR stage == COPYFILE_START Then the callback is invoked a second time with: what == COPYFILE_COPY_XATTR stage == COPYFILE_FINISH In both cases my callback function returns COPYFILE_CONTINUE. Then the copyfile function returns -1, errno is not set. Not getting a usable error unfortunately forces kind of jackhammer workarounds. Have you noticed this issue for files without resource forks (but use other xattrs) or is the resource fork the main cause?

That does the thing! Thanks! And, yes, we probably should have documented all of this, so PLEASE file about this and post the bug number back here. I filed FB22260532

I tried setting the creation date to: 1946-02-14 08:34:56 +0000 right after creating the folder. but doesn't have the desired effect. Maybe I didn't convert the date correctly?

Thanks for the info! I've been split trying to finish up a different piece of my app related to this. I briefly tried to get this to work today using the old Carbon API which I must confess I'm not super familiar with (before my time) but didn't have any success. Before I posted this thread I tried what looked like the most straightforward way to accomplish this: [fileManager createDirectoryAtURL:someURL withIntermediateDirectories:NO attributes:@{NSFileBusy:@(YES)} error:&error]; But no luck.

Thanks a lot for explaining that. It's good to know that I can do this but I'm still hesitant to use it given the 'strongly discouraged' language in the documentation. Does the 'strongly discouraged' stuff only apply to clonefile function or should I avoid 'cloning' directories in general with the copyfile function?

I just found that Quinn wrote about this a bit here: https://developer.apple.com/forums/thread/784446 I'm still unsure why the man page for one function directs you to use another function - but that man page says cloning directories is unsupported so I'm still a bit confused about the recommendation. Is cloning directories unsupported or strongly discouraged? Basically I have to move off NSFileManager because it doesn't provide progress and it doesn't support cancelling. But I'd like to clone when possible for speed and to save the user disk space.

Thanks for all the information! The other problem here is that you're assuming the APIs involved here operate on paths... when they don't. Perhaps I didn't explain myself well. I wasn't making assumptions about how the the APIs operate - the only thing I was claiming (well trying to) was that paths provide the concept of file location and in certain contexts that concept matters to GUI apps (with the NSBrowser example being an obvious one). The current API for file reference urls that I'm aware of don't try to solve that problem in any way. Note that this INCLUDES FSEvents. [...] If the directory you're monitoring moves... then FSEventStreamCreate will happily continue sending you events about that directory, NOT the original path location. If you want to know when the directory you're monitoring moves, then that's what "kFSEventStreamCreateFlagWatchRoot" Very interesting. Yeah I use that (watch root) With the watch root flag set it does seem to just track whatever is at the path. To test this I just put 'TestFolder' on my Desktop and started a FSEventStream on it. Then I rename 'TestFolder' to TestFolder2 I pick up the root change. If I start making changes in TestFolder2 I'm not picking up any changes. With the stream still open, I create a new folder 'TestFolder' - and the opened stream starts picks up changes on the new TestFolder (not TestFolder2) along the original path - so it does 'stick' to the path and doesn't follow the reference (I've never tried not using kFSEventStreamCreateFlagWatchRoot since I need it). then it might be worth starting a new thread that's focused on those issues. I've not had problems with FSEvents I just brought it up because if there was some pathless way to achieve what an FSEventStream provides I'd be open to exploring it. I'll be sure to open a new thread if I run into anything.

Revisiting this six years later - moving to copyfile instead of using NSFileManager. Nothing seems to have changed unfortunately. FB7656661 is still open.

Actually if I create the alias in my app using -bookmarkDataWithOptions:includingResourceValuesForKeys:relativeToURL:error: it resolves proper. so it appears to be an issue with the way Finder makes aliases. Maybe they are using relativeToURL: parameter? Weird.